Electronic input device, system, and method using human-comprehensible content to automatically correlate an annotation of a paper document with a digital version of the document

ABSTRACT

An electronic input device such as an electronic pen is provided to annotate a paper document. The input device records an annotation and an image of human-comprehensible content in the document sufficient to identify the document and possibly a location in the document. The human-comprehensible content is used to locate a digital version of the document and determine a corresponding location of the annotation in the digital version of the document. A computer system such as a server system may receive and store the annotation in association with the digital version of the document. The server system may further augment the digital version of the document with the annotation and send the augmented version to an output device for display and/or printing.

FIELD OF THE INVENTION

The present invention relates to systems and methods for processingannotations of a document.

BACKGROUND OF THE INVENTION

Traditionally, print media has been the only way to record informationand ideas. Thus, when a person annotated a paper version of a document,the annotation could be replicated, for example, by rewriting theannotation in another copy of the document or by making copies of theannotated document. Presently, advancements in digital imaging allowpersons to convert documents in printed form into digital form. When aperson annotates a paper version of a document, a digital version of thedocument can be made by physically scanning the annotated document intoa digital form.

Conventionally, unique identifiers have been used to identify documents.A unique identifier can be a bar code, for example, that is exclusivelyassociated with the document. Alternatively, a document may beidentified by a watermark on a page that may be created by printing aunique background pattern as a document is being printed. When a personannotates a watermarked paper version of a document, the watermarkpattern can be read by a machine to reveal the identity of the document.A watermark pattern on a page may also identify the location of theannotation in the document. However, a watermarking approach is nothelpful when an annotated document exists on paper that was notpreviously watermarked.

Conventional approaches require a person to physically scan printeddocuments with annotations after the annotations are made and to useunique identifiers such as bar codes to identify the document. Thereremains a need for better technology for automatically correlatingannotations of a paper document with a digital version of the document.The present invention addresses this need and other shortcomings in theprior art.

BRIEF SUMMARY

Exemplary embodiments of the invention described herein provide anelectronic input device, a computing system, and a computer-implementedmethod and medium for automatically correlating an annotation on a paperversion of a document with a digital version of the document usinghuman-comprehensible content in the paper version of the document. Thehuman-comprehensible content may include words, letters, characters,pictures, etc. that convey a meaning to a human reader of the document.Such content does not include unique identifiers such as bar codes,watermarks, etc., comprised of symbols that are not comprehensible to ahuman. However, if desired, the annotation can be an instructionconcerning the document or subject matter in the document.

One exemplary embodiment of the invention may include an electronicinput device such as an electronic pen. The electronic input deviceincludes a recording component that captures a user's annotation of adocument. The recording component can be an audio device that records adictation of an annotation. The recording function can also be achievedthrough a writing tip, which may include a sensor that is capable ofrecording the movements of the writing tip, and thus recording anannotation. Embodiments of the invention may further include an inkdeposit within the electronic input device. In such embodiments, theelectronic input device may contain a switch that enables or disablesthe flow of ink from the ink deposit through the writing tip.

An electronic input device may further include an imaging component forobtaining an image of human-comprehensible content in a document,wherein the human-comprehensible content is sufficient to identify alocation in the document. The imaging component can be any device thathas image recording capabilities, such as a digital camera or a scanningdevice. The electronic input device may further include a datacommunication component through which the electronic input devicecommunicates data to a computing system. In some cases, the data may beused to locate a digital version of the document that has been or isbeing annotated.

The electronic input device may further contain a processor thatprocesses the human-comprehensible content. The data resultant from theprocessing, which in some embodiments can be a hash value or a digitalsignature, may be used to locate a digital version of the document and,if desired, identify the location of the annotation in the digitalversion of the document.

Exemplary embodiments of the invention described herein also provide acomputing system for correlating an annotation on a paper version of adocument with a digital version of the document. The computing systemmay include an interface for communicating with an electronic inputdevice such as the electronic pen described above or other input device.The computing system contains or is communicatively connected to astorage containing multiple digital works. The computing system furtherincludes a processor that functions to correlate an annotation on apaper version of a document with a digital version of the documentretrieved from the storage containing multiple digital works.

If desired, images of annotated documents recorded by the electronicinput device may be subjected to an image correction process that, forexample, removes distortions in the images. Recorded images may also besubjected to a standard character recognition process that convertshuman-comprehensible content in the images into electronic text.

As previously noted, an image of human-comprehensible content in adocument may be used to locate a digital version of the document. Insome cases, the human-comprehensible content and/or spatialrelationships among the content can be used to form a digital signatureof the document. The digital signature is then used to query multiplestored digital works to find a digital version of the document with thesame or similar digital signature.

In accordance with a further aspect, the annotation is stored, forexample, as an image file, a text file, or an audio file. The annotationmay be stored in association with the digital version of the annotateddocument if a digital version of the printed document has beenidentified. In the case that the annotation is an instruction, theinstruction is recognized and executed.

In accordance with yet another aspect, a digital version of the documentmay be augmented with the annotation. The annotation may appear in thedigital version of the document at approximately the same place itappeared on the paper version of the document. The annotation may alsoappear in the digital version of the document at the bottom of a page,at the end of the document, or on a separate page, e.g., as a footnote.The augmented document may be displayed, played back, or printed uponrequest.

In another aspect, a computer-implemented method according to theinvention may include recording an annotation of a paper version of adocument. Also recorded is an image of human-comprehensible content inthe document. The image of human-comprehensible content or data derivedtherefrom may be used to locate a digital version of the document aswell as identify the location of the annotation in the document. Theannotation and location of the annotation are stored in association withthe digital version of the document. If desired, the image ofhuman-comprehensible content may be recorded at approximately the sametime that the annotation is recorded.

Embodiments of the invention are generally able to automaticallycorrelate an annotation of a paper version of a document with a digitalversion of the document using human-comprehensible content in the paperversion of the document. A user may thus annotate a paper version of adocument with the understanding that the annotation can be automaticallyassociated, stored, and reproduced with a digital version of thedocument.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing aspects and many of the attendant advantages of thisinvention will become more readily appreciated as the same become betterunderstood by reference to the following detailed description, whentaken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a pictorial diagram of a computing environment that includesan electronic input device, a server system, and an output devicecommunicably connected through one or more networks;

FIG. 2 is a depiction of an exemplary embodiment of an electronic inputdevice that may be used in the computing environment of FIG. 1;

FIG. 3 is a schematic block diagram of an exemplary embodiment of aserver system that may be used in the computing environment of FIG. 1;

FIG. 4 is a functional block diagram of a correlation program that maybe used in the server system of FIG. 3;

FIGS. 5A-5C are flow diagrams illustrating an exemplary implementationof a process for correlating an annotation of a paper document with adigital document;

FIG. 6 is a pictorial diagram illustrating one exemplary implementationof augmenting a digital work with one or more annotations; and

FIG. 7 is a pictorial diagram illustrating an alternative implementationof augmenting a digital work with one or more annotations.

DETAILED DESCRIPTION

The invention is directed to methods and apparatus, such as anelectronic input device, a computing system, and a computer-implementedmethod and medium for correlating an annotation on a paper version of adocument with a digital version of the document. More specifically, anelectronic input device is provided for recording an annotation on apaper version of a document and an image of human-comprehensible contentthat is sufficient to identify the document and possibly a locationwithin the document. The human-comprehensible content may include words,letters, characters, pictures, etc., that convey meaning to a humanreader of the document. Such content does not include unique identifierssuch as barcodes, watermarks, or any other element comprised of symbolsthat are not comprehensible to a human. The human-comprehensible contentand/or an image thereof are usable to locate a digital version of thedocument. The annotation is stored, and is associated with the digitalversion of the document if a digital version of the document isidentified. In the case that the annotation is an instruction concerningthe document or subject matter in the document, the instruction isrecognized and executed. Further aspects of the invention can be used toaugment the digital version of the document with the annotation. Thecombined result may be displayed, played back, or printed through anoutput device, for example, upon receiving a request from a user.

The following description first provides an overview of a computingsystem in which an exemplary embodiment of the invention may beimplemented. Then a method for correlating an annotation on a paperversion of a document with a digital version of the document isdescribed. The following detailed description provides exemplaryimplementations of the invention. Although specific flow diagrams andsystem configurations are illustrated, it should be understood that theexamples provided are not exhaustive and do not limit the invention tothe precise form disclosed. Persons having ordinary skill in the fieldof computers and digital imaging will recognize process steps andstructure described herein that may be interchangeable with other stepsand structure, or combinations of steps or structures, and still achievethe benefits and advantages of the invention.

To provide a context for describing embodiments of the invention, FIG. 1illustrates a pictorial block diagram of a computing environment 100that may be used to implement the invention. The computing environment100 includes at least one electronic input device 102 and a computingsystem, such as a server system 104. The electronic input device 102 maycommunicate with the server system 104 through a network 106, using, forexample, a wired or wireless connection, or may communicate directlywith the server system 104 without an intervening network. Asillustrated in FIG. 1, the server system 104 may be associated with amass storage that contains a digital works database 108 and anannotations database 110. These two databases may be part of the serversystem 104; they may also be remotely located and communicativelyconnected to the server system 104.

As will be described with regard to the particular embodiments shownherein, the server system 104 is configured to communicate with theelectronic input device 102. For example, the server system 104 mayreceive an annotation that was made on a paper version of a document andan image of human-comprehensible content in the document that issufficient to identify the document and possibly a location in thedocument. In some cases, the electronic input device 102 processes theimage of human-comprehensible content and generates data, such as a hashkey or a digital signature. In other cases, the human-comprehensiblecontent itself is used to identify a digital version of the document andpossibly a location in the document. The electronic input device may,but is not required to, process the image of human-comprehensiblecontent before sending it to the server system 104.

The server system 104 uses the data received to query the digital worksdatabase 108 for a digital version of the document that is being or hasbeen annotated. The server system 104 stores the annotation in theannotations database 110, in association with a digital version of thedocument if a digital version of the document has been found. Incircumstances where the server system 104 recognizes some or all of theannotation as constituting an instruction, the server system 104 alsoexecutes (or causes to be executed) the instruction portion of theannotation. As will be described later herein, in some embodiments ofthe invention, the digital version of the document may be augmented withthe annotation. The resultant combination may be sent to an outputdevice 112 through the network 106.

The output device 112 shown in FIG. 1 communicates with the serversystem 104 through the network 106. An output device 112 incommunication with the server system 104 may include typical outputdevices, such as a computer display (e.g., CRT or LCD screen), atelevision, printer, facsimile machine, copy machine, etc. As notedabove, the output device 112 can be used to display, play back, and/orprint a digital version of a document augmented with an annotationprovided to the server system 104 by the electronic input device 102.

Digital works may include any type of content that may be stored ordistributed in digital form. By way of illustration, without limitation,digital works can include all forms of textual and graphicalinformation, such as books, magazines, newspapers, newsletters, menus,guides, references, photographs, articles, reports, documents, etc., andall forms of audio and audio visual works such as music, multimediapresentations, audio books, movies, etc. Digital works may be stored asmedia files in the digital works database 108 that resides in or isotherwise communicatively connected to the server system 104.

An annotation can be any form of content that is associated with adocument. It may include (but is not limited to) a critical commentary,an explanatory note, a summary of the document or some aspect of thedocument, a separate explanation of the same topic presented in thedocument, or a simple mark to highlight or emphasize something in thedocument. Annotations may be words, graphs, formulae, files, images,enhancements, markings, etc., provided by a user. For example, a usermay wish to annotate a travel book by providing recent images oflocations described in the travel book. A user may annotate a musicsheet by commenting on the music, providing background information onthe composer, alternative lyrics, technical information as to thecomposition, related musical compositions, etc. In a circumstance wherethe document is a cookbook, for example, a user may identify alternativeingredients that enhance a recipe, ingredients that make a spicier dish,ingredient substitutions considered to be lower in fat or carbohydrates,etc. In another circumstance, a user such as a student or teacher mayprovide a markup of an academic textbook. As can be appreciated from theforegoing, there is virtually no limitation to the type, form andcontent of annotations that can be added to a digital work. By way ofillustration, an annotation can also be in the form of a link or addressto another page, file, or document available to the server system 104 orother computing system.

In embodiments of the invention, an annotation can also be aninstruction concerning the document or subject matter in the document.For example, an annotation can be a command to look up a certain word inthe document, to print the document, to check existing commentariesconcerning the content of the document, to print a picture of a selectedobject in the document, etc. An annotation can also be a command toroute the annotated document to a specified location, to generate anotification to be sent to a mail list to notify members on the maillist, for example, that the document has new annotation(s), etc. Anannotation can further be a command to execute an action on a remotesystem. For example, an annotation can be a command to generate ane-mail in a remote server; the e-mail attaches the annotated document.In circumstances where the annotation is written, a characterrecognition routine, such as OCR, may be applied to the annotation; andif one or more characters or words in the annotation indicate anexecutable instruction, the instruction can then be carried out. Theexecution of the instruction may occur locally on the electronic inputdevice, remotely on the server where the annotation is sent, or remotelyon another computing system that is specified in the instruction.

The computing environment 100 illustrated in FIG. 1 has an annotationsdatabase 110 residing in or otherwise communicatively connected to theserver system 104, though in other computing environments, the computingsystems may be arranged differently. Annotations that are stored in theannotations database 110 may be associated with the digital works storedin the digital works database 108. The network 106 shown in FIG. 1 maybe a local area network (LAN) or a larger network, such as a wide areanetwork (WAN), a collection of networks, or the Internet. Wired orwireless network connections may be used with any suitable communicationprotocols, such as TCP/IP. As will be appreciated by those skilled inthe art and others, FIG. 1 provides a simplified example of one suitablecomputing environment for implementing embodiments of the invention, andis not limited thereto.

When software and hardware formed in accordance with the invention isimplemented in one or more input devices and computing systems of thetype illustrated in FIG. 1, for example, the devices and systems providea way for users to annotate a paper version of a document such that theannotation can be automatically associated with a digital version of thedocument. In embodiments of the invention, the devices and systems mayalso provide a way for augmenting the digital version of the documentwith the annotation and present the augmented document for further use.Allowing an annotation on a paper version of a document to be associatedwith a digital version of the document and/or providing a mechanism foraugmenting a digital version of a document with an annotation can makethe underlying digital work more useful and valuable. For example, uponrequest, the digital version of the document can be printed along withthe annotation.

FIG. 2 provides one exemplary implementation of an electronic inputdevice 102 that can be used to correlate an annotation on a paperversion of a document with a digital version of the document. Theelectronic input device contains a recording component that enables auser to annotate a document. In some embodiments of the invention, therecording component is an audio device 201. When the audio device 201 isturned on, a user may dictate an annotation, which will be recorded inthe audio device 201. In other embodiments of the invention, therecording component is a writing tip 202 for a user to annotate adocument. The writing tip 202 may include a sensor, having a capacity torecord the movements of the writing tip 202. The electronic input device102 may further include an ink deposit. In some embodiments, theelectronic input device 102 contains a switch that enables or disablesthe flow of ink from the ink deposit through the writing tip 202. Theswitch, for example, may control a valve or other mechanism that canopen and close a channel through which ink from the ink deposit mayflow.

In embodiments that include a sensor with the writing tip 202, thesensor captures the annotation by recording the movements of the writingtip 202. Any form or type of sensor may be used to record the movementsof the writing tip 202. Some examples, without limitation, include: asolid state gyro that is configured to detect and report motion of thewriting tip 202, e.g., in the X-Y plane, as the writing tip moves acrossthe document being annotated; a roller ball that detects and reportsmotion of the writing tip 202, e.g., in a manner similar to a rollerball in a conventional computer mouse; an RF signal that emanates fromthe electronic input device 102 from which the relative motion of thewriting tip 202 can be determined; or a digitizer positioned around orunder the document page being annotated. At its simplest, the sensorneeds only to detect and to provide signals reflecting the relativemotion of the writing tip 202 as it moves across the document beingannotated.

The electronic input device 102 may further contain an imaging component204 for obtaining an image of human-comprehensible content in thedocument that is sufficient to identify the document and possibly alocation in the document. The imaging component 204 may be any devicethat has image recording capabilities. For example, the imagingcomponent 204 can be a digital camera, a scanning device, etc. Theimaging component may be configured to record a single image, or recorda series of images, that capture human-comprehensible content in thedocument. The human-comprehensible content is preferably adjacent to orspatially near the annotation, though embodiments of the inventions maybe configured to operate using human-comprehensible content locatedanywhere in the document. As will be understood below, thehuman-comprehensible content is used to identify the digital version ofthe document that is being or has been annotated. Where thehuman-comprehensible content is adjacent to or spatially near (includingunder) the annotation, the content can be used to identify the locationin the digital version of the document where the annotation is found inthe printed version of the document. The imaging component can also beused to record the annotation itself.

Finally, the electronic input device 102 includes a data communicationcomponent 206 through which the electronic input device 102 communicateswith a computing system, such as the server system 104 depicted inFIG. 1. The data communication component 206 is used to upload the dataand information recorded by the imaging component 204 and by therecording component, which can be the audio device 201 and/or thewriting tip 202.

If desired, the electronic input device 102 records and transmitsinformation while the user is annotating. In other embodiments, theelectronic input device records and transmits information after theannotation has been completed. The data communication component 206hosts a wired or wireless connection unit through which the electronicinput device 102 communicates.

In some embodiments of the invention, the electronic input device 102includes a processor that processes human-comprehensible content togenerate data such as a hash value or a digital signature concerning thehuman-comprehensible content as will be described in greater detailherein. Such data is then transmitted to a computing system, such as theserver system 104 illustrated in FIGS. 1 and 3.

In some embodiments of the invention, the electronic input device 102may be associated with a docking station. In such embodiments, theelectronic input device 102 captures and stores data, such asannotations and images of human-comprehensible content in a documentthat is sufficient to identify the document and possibly a location inthe document. The electronic input device 102 communicates theannotations and images of the human-comprehensible content to acomputing system, such as the server system 104 illustrated in FIGS. 1and 3, when the electronic input device 102 is docked in the dockingstation.

FIG. 3 depicts an exemplary computer architecture for the server system104 illustrated in FIG. 1, suitable for use in implementing embodimentsof the invention. The server system 104 connects to the network 106(FIG. 1) using a network interface 302. The network interface 302includes the necessary hardware and software to allow the server system104 to communicate with other computing devices connected to the network106 by use of one or more suitable communication protocols, such asTCP/IP protocol.

The server system 104 further includes a processor 304, memory 306, andan input/output interface 308, all communicatively connected togetherand to the network interface 302 by a communication bus 310. Theprocessor 304 is configured to operate in accordance with computerprogram instructions stored in a memory, such as the memory 306. Programinstructions may also be embodied in a hardware format, such as aprogrammed digital signal processor.

The memory 306 may be comprised of any type of storage facility, and mayinclude, by way of example, RAM, ROM, and/or removable computer-readablemedia. The memory 306 may store an operating system 312 for controllingthe operation of the server system 104. The operating system 312 may bea general-purpose operating system such as a Microsoft server operatingsystem, UNIX, or LINUX, for example.

The memory 306 additionally stores program code and data that provide adatabase manager 314 and a correlation program 316. The correlationprogram 316 comprises computer-executable instructions that, whenexecuted by the processor 304, cause the server system 104 to correlatean annotation on a paper version of a document with a digital version ofthe document. The correlation program 316 receives the annotation froman electronic input device such as the electronic input device 102illustrated in FIGS. 1 and 2 through the input/output interface 308. Thecorrelation program 316 may also cause the server system 104 to augmentthe digital version of the document with the annotation and output theresultant combination to an output device 112 through the input/outputinterface 308. Furthermore, the memory 306 may be configured to storedigital images of content and data for processing, transmission, anddisplay in accordance with the invention.

In this particular embodiment, the server system 104 further includes amass storage 320 comprising facilities such as one or more CD-RW/DVD-RWdrives, hard disk drives, tape drives, etc., or combinations thereof,that are utilized to store databases. In the embodiment shown in FIG. 3,the mass storage 320 contains the digital works database 108 and theannotations database 110. The database manager 314 in the memory 306 canbe used to facilitate the storage and retrieval of information in thedigital works database 108 and the annotations database 110.

For ease of illustration, FIG. 3 does not show other typical componentsof a computing system, such as a video display adapter, power supply,computer monitor, etc. However, those having ordinary skill in the artof computers will recognize a wide selection of commercially-availablecomponents that can be used to construct and operate a computing systemsuch as the server system 104 illustrated in FIG. 3.

As noted above, the server system 104 may receive data concerning anannotation on a paper version of a document from an electronic inputdevice, such as the electronic input device 102 illustrated in FIGS. 1and 2. The annotation is stored in the annotations database 110, and maybe stored in association with a digital version of the document if adigital version of the document is found. Storing an annotation inassociation with a digital version of the document may be accomplishedin any suitable manner. Such a manner can be, for example, simplystoring the annotation in the annotations database 110 with anindication of the digital work to which the annotation pertains, and ifappropriate, the location(s) within the digital work to which theannotation pertains.

FIG. 4 provides a functional block diagram of one exemplaryimplementation of the correlation program 316 illustrated in FIG. 3. Asnoted above, the correlation program 316 receives information about anannotation of a document and searches to locate a digital version of thedocument in the digital works database 108. If a digital version of thedocument is found, the correlation program 316 saves the annotation inthe annotations database 110 in association with the digital version ofthe document. If the digital version of the document is not found, thecorrelation program 316 still saves the annotation in the annotationsdatabase 110, expecting a digital version of the document can beautomatically or manually identified at a later time. In someembodiments of the invention, the correlation program 316 is furthercapable of augmenting the digital version of the document with theannotation and providing the resultant combination for display,playback, and/or printing to an output device such as the output device112 illustrated in FIG. 1.

FIG. 4 specifically illustrates exemplary components that may beincluded in a correlation program 316. These components include an imagecorrection module 402, a character recognition module 404, and a sourceidentification module 406. Collectively, the components of thecorrelation program 316 can be used to correlate an annotation on apaper version of a document with a digital version of the document.

As described above with respect to FIGS. 1 and 2, the electronic inputdevice 102 records an annotation and an image of human-comprehensiblecontent in a document (hereinafter “comprehensible content”) that issufficient to identify the document and possibly a location in thedocument. The comprehensible content may include words, letters,characters, pictures, etc. that convey meaning to a human reader of thedocument. Such content does not include identifiers, such as bar codes,watermarks, cross hatches, etc., that are not comprehensible to a human,even though they can be used by a machine to uniquely identify adocument. Embodiments of the invention can be configured to use thecomprehensible content and possibly spatial relationships among thecomprehensible content to locate the digital version of the document, aswill be discussed later below in regard to FIG. 5C.

In some embodiments of the invention, the electronic input device 102may record and transmit an annotation and an image of comprehensiblecontent in a document before, during or after the time in which thedocument is annotated. If the data contains an image of the annotationand/or comprehensible content in the document, upon receiving the image,the image correction module 402 may be engaged to reduce or eliminatedistortions, if any, in the image. Any of a variety of known methods forimage correction may be used in conjunction with the invention. Forexample, the image correction module 402 may be comprised of one or moreimage correction applications such as a perspective skew correctionapplication, a rotational skew correction application, and a movementstabilization application. Individually or collectively the applicationscomprising the image correction module 402 can operate to reduce oreliminate distortions in images so that a character recognitionapplication may better identify comprehensible content, including text,in the corrected image.

One example of a perspective skew correction application calculates theperspective skew angles of images and adjusts the images to reduce oreliminate perspective skew. The angles of perspective skew can becalculated by analyzing one or more indictors that exist in the imagesof printed content. For example, a linear indicator of the skew anglesis the predominant direction of white space on a page. Typically, whitespace exists between successive lines of text and the orientation of apage in relation to an input device may be derived by calculating theprevailing direction of white space. This linear indicator and otherindicators such as the prevailing direction of character strings,orientation of geometric shapes and page edges, and the orientation ofindividual characters may all be used, individually or collectively, todetermine the perspective skew angles of a page image.

After the perspective skew angles are calculated, a perspective skewcorrection application can adjust the images to reduce or eliminateperspective skew. More specifically, the perspective skew correctionapplication can map pixels from a source image to a destination imagebased on a mapping function that eliminates perspective skew. The sourceimage may be mapped to a destination image with a technique known asmesh warping transformation. Generally described, mesh-warpingtransformation applies two-dimensional geometric transformations toreshape geometric alignments. In order to reshape geometric alignments,a correspondence is established between feature primitives, e.g.,curves, line segments, and points. Each feature primitive specifies animage feature or landmark. The correspondence between feature primitivesand the perspective skew angles is then used to compute a mappingfunction that defines the change in each pixel's location between thesource and destination images. Applying the mapping function to thesource image produces a destination image free from perspective skew.One algorithm for performing a mesh warping transformation that may beused in conjunction with the present invention to eliminate perspectiveskew may be found in “A Mesh Warping Algorithm Based on WeightedLapacian Smoothing,” by Suzanne Shontz and Stephen Vavasis, inProceedings of the 12th International Meshing Roundtable, SandiaNational Laboratories, September 2003, pp. 147-158.

Rotational skew refers to distortion in an image when text follows acurved path out of alignment with an associated line of text. Rotationalskew in a page image is common when text is located near a binding. Arotational skew correction application can determine if an imagecontains text that is rotationally skewed and perform adjustments toreduce or eliminate the skew. To determine if rotational skew exists,one embodiment of a rotational skew correction application analyzeslines of text and determines the predominant direction of the text. Textthat is out of alignment with the predominant direction and follows acurved path is identified as being rotationally skewed. Similar to theperspective skew correction application described above, the rotationalskew correction application may use a technique of mesh warpingtransformation to produce an image that is not rotationally skewed.However, the mapping function that corrects rotational skew generallyuses different parameters than the mapping function that correctsperspective skew. For example, the mapping function generated by therotational skew correction application determines the amount that textis rotationally skewed (i.e., rotational variance). Then acorrespondence is established between feature primitives. Thecorrespondence between feature primitives and the rotational varianceare then used to calculate the mapping function that corrects rotationalskew. One algorithm for performing a mesh warping transformation thatmay be used in conjunction with the present invention to eliminaterotational skew may be found in “A Mesh Warping Algorithm Based onWeighted Lapacian Smoothing,” by Suzanne Shontz and Stephen Vavasis,referenced above.

The imaging component 204 of the electronic input device described abovemay capture one or more images of the document with text that isdistorted or blurred due to movement. A movement stabilizationapplication can adjust an image to reduce or eliminate this type ofdistortion. More specifically, a movement stabilization application canidentify character edges in images by identifying pairs of adjacentpixels that have a predetermined difference in brightness. Thebrightness of each pixel along an edge is then exaggerated so that darkpixels are darkened and light pixels are lightened. The result is toincrease the contrast between neighboring pixels and create characterswith visually perceptible edges.

It should be well understood that the image correction module 402 mayimplement other techniques for correcting images that are generallyknown in the art. For example, character recognition applications oftenhave difficulty recognizing text in images with heavy noise (i.e.,grainy or heavily pixilated images). The image correction module 402 mayreduce the amount of noise in an image by adjusting pixels that are notassociated with text in the image.

The character recognition module 404 operates on the image to identifycomprehensible content, such as text, in the image. Where suitable, thecharacter recognition module 404 may translates images of charactersusing a standard encoding scheme (such as ASCII or Unicode) to digitallyrepresent the characters in an image. These standard encoding schemesspecify a correspondence between digital bit patterns and symbols of awritten language, thus allowing a computer to process, store andcommunicate character-oriented information. In embodiments of theinvention, the term “character recognition” refers to all forms ofoptical character recognition (using optical techniques such as mirrorsand lenses) and digital character recognition (using scanners andcomputer algorithms). Since very few applications remain that use trueoptical techniques, the term OCR as used herein includes all forms ofoptical and digital character recognition.

In this particular embodiment, text that is identified by the characterrecognition module 404 may constitute comprehensible content in thedocument that is usable to locate the digital version of the document.As will be described below, the source identification module 406 is thenengaged to query the digital works database 108 to identify a digitalversion of the document that is being or has been annotated. The sourceidentification module 406 may also determine the location in the digitalversion of the document that corresponds to the location of theannotation in the paper version of the document.

Finally, the correlation program 316 works in conjunction with thedatabase manager 314 to store the annotation in the annotations database110, preferably identifying the location at which the annotation shouldappear in the digital work. In some embodiments of the invention, thecorrelation program 316 may further be employed to augment the digitalwork with the annotation and provide the augmented digital work fordisplay, play back, and/or printing to an output device, such as theoutput device 112 illustrated in FIG. 1.

FIGS. 5A-5C illustrate one exemplary implementation of a process 500 forcorrelating an annotation on a paper version of a document with adigital version of the document. The process 500 is described withreference to the systems and programs illustrated in FIGS. 1-4. Brieflystated, the process 500 receives from an electronic input device 102 animage of human-comprehensible content in a document in a document thatis sufficient to identify the document and possibly a location in thedocument. The process 500 uses the data to locate a digital version ofthe document in the digital works database 108. Upon locating a digitalversion of the document, the process 500 further identifies the locationof the annotation in the digital version of the document. The locationof the annotation may be associated with any aspect of the document,including but not limited to the front or back of the document or asection, chapter, page, paragraph, sentence, phrase, word, character,etc. in the document. The process 500 then stores the annotation in theannotation database 110.

As noted above with reference to FIG. 2, in some embodiments of theinvention, the electronic input device 102 records images ofcomprehensible content in a document and uploads the images to theserver system 104, as indicated at block 502. The process 500 executes aroutine 504 to locate a digital version of the document in the digitalworks database 110, using the received images. FIG. 5C illustrates oneexemplary implementation of the routine 504 and will be discussed indetail below.

After executing the routine 504, the process 500 determines whether adigital version of the document has been found, as indicated at decisionblock 506. If the process 500 has not been able to locate a digitalversion of the document using the received image(s), the process 500proceeds to check if it has received additional images containingcomprehensible content from the electronic input device 102. Seedecision block 508. If the answer is YES, the process 500 loops back tothe routine 504 to search for a digital version of the document usingthe additional images. If the process 500 does not receive additionalimages from the electronic input device 102, the process 500 proceeds toreceive and store the annotation from the electronic input device 102.See block 509. A digital version of the document may be identifiedlater, by manual or automatic searching and matching.

If the answer to decision block 506 is YES, the process 500 hassuccessfully located a digital version of the document, and proceeds atcontinuation terminal A (FIG. 5B) to identify the location of theannotation in the digital version of the document. See block 510. Insome embodiments of the invention, the process 500 may also send anotification to the electronic input device 102, acknowledging thesuccessful identification of a digital version of the document that theuser is annotating, as indicated at block 512. For example, theelectronic input device 102 may emit a beep upon receiving thenotification from the server system 104. At that moment, the electronicinput device 102 may stop transmitting images used for purposes ofidentifying the digital version of the document, though additionalimages containing one or more annotations on the paper version of thedocument may be sent to the server system 104.

At block 514, the process 500 receives the annotation and proceeds toprocess the annotation, as indicated at block 516. As noted above, theannotation can be (or include) an executable instruction concerning thedocument. Such an instruction can be, for example, to look up a certainphrase or word in the document, to locate annotations made by otherusers, to locate annotations concerning a certain subject matter in thedocument, to print an object in the document, to execute an action on aremote system, etc. Typically, an annotation containing an instructionstarts with a one or more predefined characters or words that can berecognized as indicating an executable instruction. Upon identifyingsuch an annotation with an instruction, the process 500 executes theinstruction contained in the annotation. Alternatively, the process 500may trigger a new process to execute the instruction.

Part of processing the annotation at block 516 may include storing theannotation in the annotation database 110. In some embodiments of theinvention, the process 500 receives and stores the annotation as animage file. In other embodiments, the process 500 processes theannotation using a standard character recognition process that convertsthe annotation into text, and stores the annotation as a text file. Atext file may also be used to store the annotation when the annotationis received in text format from the user (e.g., using a keyboard).Alternatively, in the case that the annotation was dictated by the userand captured by the audio device 201 in the electronic input device 102,the annotation can be saved as an audio file.

In some embodiments of the invention, the process 500 augments a digitalversion of the document with the annotation, as indicated at block 518.When augmenting a digital version of a document with an annotation, theprocess 500 may cause the annotation to appear in the digital version ofthe document at approximately the same location as the annotationappeared on the paper version of the document, otherwise referred toherein as in-line insertion. In other embodiments, the process 500 maycause the annotation to appear as a footnote to the digital version ofthe document on the page where the annotation is located, at the end ofthe document, or on a separate page. If desired, a footnote indicatormay appear in the digital version of the document at approximately thesame location that the annotation appeared on the paper version of thedocument. The annotation may be presented in a graphic format (typicallyas an image of handwriting) or in a text format (which may be the resultof applying OCR technology to a handwritten annotation or input receivedfrom a device such as a keyboard). A handwritten annotation may bedisplayed just as it is received from the electronic input device 102.The process 500 may then send the augmented document to an outputdevice, such as the output device 112 illustrated in FIG. 1, asindicated at block 520. It should be understood, in the context of theinvention, the terms “text” and “character” include all forms ofletters, characters, symbols, numbers, formulae, etc., that may be usedto represent information on a paper version of a document. The term“page” broadly refers to any space of two dimensions or greater in whichcontent may reside.

FIG. 5C illustrates one exemplary implementation of the routine 504referenced in FIG. 5A for locating a digital version of a document. Theroutine 504 implements aspects of the correlation program 316illustrated in FIG. 4. Upon receiving an image from the electronic inputdevice 102, the routine 504 first checks at decision block 530 if thereis any distortion in the image received. If there is distortion in theimage, the routine 504 proceeds to block 532 to correct the image usingan image correction module, such as the image correction module 402illustrated in FIG. 4 and described above. In a case where there is nodistortion in the image or if the distortion has been corrected (or ifno distortion correction is available or desired), the routine 504proceeds to block 534 to process the image. When processing the image,the routine 504 may perform a character recognition process on theimage. As described above, with respect to the character recognitionmodule 404 illustrated in FIG. 4, a character recognition processconverts characters in an image to electronic text. Examples ofcommercially-available optical character recognition software includeOmniPage Pro from ScanSoft, Inc., and FineReader from SmartLinkCorporation.

The routine 504 then proceeds to block 536 to locate a digital versionof the document using the human-comprehensible content in the document.In some embodiments of the invention, one or more types ofcomputer-implemented matching may be used to identify a digital versionof the document. Such computer-implemented matching can include textmatching using text resulting from performing character recognition onthe received image. Alternatively, in other embodiments, signaturematching using a digital signature derived from the received image maybe used to identify a digital version of the document.

For example, using one or more strings of text (i.e., a collection ofone or more characters) from the document obtained via the characterrecognition process, the server system 104 may search a database ofcontent of known source for strings of content that match. In thisexample, the server system 104 may compare some or all of the textobtained from the printed document with digital text stored in thedigital works database 108. If matching text is found in the digitalworks database 108 to uniquely correlate the image of comprehensiblecontent with content of a known source, the identity of the digitalversion of the document is known. Similar matching algorithms may beemployed where document is an audio or video work and the “image”received from the electronic input device 102 is a segment of audio orvideo. The server system 104 may compare the segment of audio or video,or some transformation thereof, with a digital works database of knowncontent in order to find content that matches the input segment. Ifmatching content is identified, the identity of the digital version ofthe document is known.

Other forms of computer-implemented matching may be used in embodimentsof the invention. For example, rather than using a string of text tosearch the digital works database 108, the image of the document mayitself be used to generate a digital signature that is then compared todigital signatures of content of known source. As will be seen below,optical character recognition may or may not be needed in suchembodiments.

In one embodiment, a digital signature is generated from an image of adocument by quantifying one or more characteristics of the image.Characteristics such as the amount of white space existing on a page,the distance between lines of printed text, the number of characters,words, or ink marks on the page or in a segment of the text in the page,or the placement of graphics in text or in the page, may all be used,either individually or collectively, to generate a digital signaturethat represents the image. Signal information in an image may also betransformed to another domain, such as a frequency domain, to generate adigital signature that represents the image.

Once a digital signature is generated from the image of the document,the digital version of the document may be identified by searching arepository of digital signatures generated from digital works of knownsource using the same process as employed on the received image. Amatching digital signature results in identification of the digitalversion of the imaged document. Some caution should be exercised howeverwhen generating and comparing digital signatures of images withdistortions from faxing, scanning, multi-generation copying,inconsistent lighting or shadowing, or other skewing that may change ordegrade the quality of an image of content.

In yet another embodiment, a digital signature may be obtained using ahash function that operates on a subset of the comprehensible content,such as text, obtained from an image of a document. The hash functionproduces a hash key (sometimes referred to as a hash value). As known tothose skilled in the art, hash functions can be used to represent alarge amount of data with a smaller range of data, and are particularlysuitable for fast searching. In accordance with one exemplaryembodiment, a hash key for a page of printed content may be generatedusing digital text contained within predetermined fields on the page.For example, the first character of each line of text in a page and thelength of the first word in each line may be combined to form a hashkey. In another example, a hash key may be generated using a hashfunction that identifies the first character in one or more lines oftext and combines it with the number of “ink marks” or strokes used inthe first word, including punctuation, in each line of text.

Multiple hash keys for an image of a document may also be derived andused for searching the digital works database 108 to identify thedigital version of the document. For example, a hash function could beemployed on portions of text occurring at the top and bottom and leftand right corners of the text. Having multiple hash keys for a documentimage is useful in cases where part of an image received from theelectronic input device 102 is obscured or distorted. A hash functionoperating on text in an unobscured or undistorted portion of the imagemay be used to generate a suitable hash key that identifies the digitalversion of the document when a matching hash key is found in the digitalworks database 108. In these examples, the optical character recognitionprocess is not necessarily performed on an entire image of a documentnor is an image containing an entire page of a document required.

After identifying the digital version of the document, the routine 504may also identify the location of the annotation in the document, asindicated at block 538, for later display of the annotation with thedigital version of the document. In one exemplary implementation, theroutine 504 may incorporate data obtained from the character recognitionprocess indicating the relative location of text identified in the imagereceived from the electronic input device 102. OCR software is capableof recording X-Y positions of text identified in an image. One or moresegments of identified text adjacent to an annotation in the image maybe selected by the routine 504. The digital version of the document isanalyzed and the location of these segments in the digital version ofthe document is determined. Since the selected segments were adjacent to(or under) the annotation, the location of the annotation in the digitalversion of the document is known. This location information can bestored with the annotation in the annotations database 110. One or moredistances may also be calculated between the selected segments and theannotation itself. The additional distance information can be used toposition the annotation more precisely with the digital version of thework when augmenting the digital work with the annotation at a latertime.

Returning to block 536 described above, OCR text position data can alsobe used to identify the digital version of the document. Segments oftext identified in an image received from the electronic input device102, and the relative distance between the segments, can be determinedto form a unique digital signature. Documents of known identity in thedigital works database 108 (stored with text position data possiblyobtained by the same OCR process run earlier on the known documents) areanalyzed to see if any one of them contains the same segments of texthaving the same spatial relationship among them. When a document isfound with the same segments of text having the same spatialrelationship among them (i.e., a matching digital signature), theidentity of the document is known.

Alternatively, as noted above with reference to FIG. 2, in someembodiments of the invention, the electronic input device 102 processesthe image of the human-comprehensible content to generate data such as ahash key or a digital signature concerning the human-comprehensiblecontent. In these embodiments, the electronic input device 102 maygenerate a hash key or digital signature by using the same or similarhash key and digital signature generation procedures described above.Upon receiving such data, the process 500 may use the one or more typesof computer-implemented matching described above to identify a digitalversion of the document and the location of the annotation in thedigital version of the document.

As for augmenting and presenting the digital version of the documentwith stored annotations, as previously described, the annotation mayappear in the digital version of the document at approximately the samelocation where the annotation appeared on the paper version of thedocument. In one embodiment, a layered display may be used to createthis effect where the digital work (without annotation) is displayed andthe annotation is displayed on another layer superposed over the layerwith the digital work. Alternatively, the data forming an image of thedigital work (e.g., bit map or vector data) may be modified toincorporate the data representing the annotation, with the modifieddigital work then being provided for display. FIG. 6 illustrates adigital version of a annotations 602 and 604 may be inserted as imagecomponents or textual components. In other embodiments, annotations maybe added to the digital version of the document as wherein theannotations 602 and 604 are appended to the digital work 600 as footnote1 and footnote 2. Footnote indicators (here shown as FN1 and FN2) may beinserted as image or textual components in the display of the digitalwork to show where the annotations appeared in the paper version of thedocument.

While several embodiments of the invention have been illustrated anddescribed, it will be appreciated that various changes can be madetherein without departing from the spirit and scope of the invention.The scope of the invention, therefore, should be determined from thefollowing claims and equivalents thereto.

1. An electronic input device for use in annotating a document,comprising: (a) an imaging component for obtaining an image ofhuman-comprehensible content in a paper version of a document sufficientto identify a location in the document; (b) a recording component forrecording an annotation to be associated with the identified location inthe document; (c) a recognition component configured to analyze theannotation and determine if the annotation includes anelectronically-executable instruction; and (d) a data communicationcomponent configured to communicate data from the electronic inputdevice to a computer system.
 2. The electronic input device of claim 1,wherein the recording component is a writing tip of the electronic inputdevice.
 3. The electronic input device of claim 2, wherein the writingtip includes a sensor for recording movements of the writing tip.
 4. Theelectronic input device of claim 3, wherein the sensor records movementsof the writing tip in a two-dimensional X-Y plane.
 5. The electronicinput device of claim 2, further comprising an ink deposit connected tothe writing tip.
 6. The electronic input device of claim 5, furthercomprising a switch for enabling or disabling the flow of ink from theink deposit through the writing tip.
 7. The electronic input device ofclaim 1, wherein the recording component includes an audio device thatis capable of capturing an audible annotation.
 8. The electronic inputdevice of claim 1, wherein the imaging component includes a camera. 9.The electronic input device of claim 1, wherein the imaging componentrecords a spatial relationship among the human-comprehensible content inthe document.
 10. The electronic input device of claim 1, furthercomprising a processor for processing the human-comprehensible contentin the image obtained by the imaging component.
 11. The electronic inputdevice of claim 1, wherein the electronically-executable instructionstarts with one or more predefined characters or words that arerecognized as indicating the executable instruction.
 12. A computingsystem for correlating an annotation of a paper version of a documentwith a digital version of the document, comprising: (a) an interface forcommunicating with one or more input devices; (b) a storage containingdigital versions of documents; and (c) a processor in communication withthe interface and the storage, wherein the processor is configured to:(i) receive through the interface, from an input device, an image ofhuman-comprehensible content in a document, wherein thehuman-comprehensible content is sufficient to identify a digital versionof the document; (ii) use the image of human-comprehensible content tolocate in the storage a digital version of the document; (iii) receivethrough the interface, from the input device, an annotation of a paperversion of the document; (iv) apply a recognition routine to theannotation and determine if the annotation includes anelectronically-executable instruction; and (v) store the annotation inassociation with the digital version of the document.
 13. The computersystem of claim 12, wherein the processor is configured to locate adigital version of the document using a digital signature derived fromthe image of human-comprehensible content.
 14. The computer system ofclaim 12, wherein the processor is configured to locate a digitalversion of the document using a hash key generated from the image ofhuman-comprehensible content.
 15. The computer system of claim 12,wherein the processor is configured to locate a digital version of thedocument using the human-comprehensible content in the image.
 16. Thecomputer system of claim 12, wherein the processor is configured tolocate a digital version of the document using a spatial relationshipamong the human-comprehensible content in the image.
 17. The computingsystem of claim 16, wherein the processor is further configured to usethe spatial relationship to identify the location of the annotation inthe document.
 18. The computing system of claim 12, wherein theprocessor is configured to store the annotation as any one of a textfile, an audio file, or an image file.
 19. The computing system of claim12, wherein the processor is further configured to execute theelectronically-executable instruction.
 20. The computing system of claim12, wherein the processor is further configured to augment the digitalversion of the document with the annotation.
 21. The computing system ofclaim 20, wherein the processor is configured to cause the annotation toappear in the digital version of the document at approximately the samelocation as the annotation appears on the paper version of the document.22. The computing system of claim 20, wherein the processor isconfigured to cause the annotation to appear as a footnote to thedigital version of the document with a footnote indicator appearing atapproximately the same location as the annotation appears on the paperversion of the document.
 23. The computing system of claim 20, whereinthe processor is further configured to print the augmented digitalversion of the document.
 24. The computer system of claim 12, whereinthe electronically-executable instruction starts with one or morepredefined characters or words that are recognized as indicating theexecutable instruction.
 25. A computer-implemented method for processingan annotation of a document, comprising: (a) receiving an image ofhuman-comprehensible content in a paper version of a document, whereinthe human-comprehensible content is sufficient to identify a digitalversion of the document; (b) processing the human-comprehensible contentto locate a digital version of the document; (c) receiving an annotationof the paper version of the document; (d) determining if the annotationincludes an electronically-executable instruction; and (e) storing theannotation in association with the digital version of the document. 26.The computer-implemented method of claim 25, wherein receiving theannotation includes receiving an image of the annotation as it appearson the paper version of the document.
 27. The computer-implementedmethod of claim 25, wherein processing the human-comprehensible contentproduces a digital signature that is usable to locate the digitalversion of the document.
 28. The computer-implemented method of claim25, wherein processing the human-comprehensible content produces a hashkey that is usable to locate the digital version of the document. 29.The computer-implemented method of claim 25, wherein processing thehuman-comprehensible content uses the human-comprehensible contentitself to locate the digital version of the document.
 30. Thecomputer-implemented method of claim 25, further comprising determininga spatial relationship among the human-comprehensible content and usingthe spatial relationship to locate the digital version of the document.31. The computer-implemented method of claim 30, further comprisingusing the spatial relationship to identify the location of theannotation in the document.
 32. The computer-implemented method of claim25, wherein storing the annotation includes storing the annotation asany one of a text file, an audio file, or an image file.
 33. Thecomputer-implemented method of claim 25, further comprising executingthe electronically-executable instruction.
 34. The computer-implementedmethod of claim 25, further comprising augmenting the digital version ofthe document with the annotation.
 35. The computer-implemented method ofclaim 34, wherein augmenting the digital version of the document withthe annotation includes causing the annotation to appear in the digitalversion of the document at approximately the same location as theannotation appears on the paper version of the document.
 36. Thecomputer-implemented method of claim 34, wherein augmenting the digitalversion of the document with the annotation includes causing theannotation to appear as a footnote in the digital version of thedocument with a footnote indicator appearing at approximately the samelocation as the annotation appears on the paper version of the document.37. The computer-implemented method of claim 34, further comprisingprinting the augmented digital version of the document.
 38. Thecomputer-implemented method of claim 25, wherein theelectronically-executable instruction starts with one or more predefinedcharacters or words that are recognized as indicating the executableinstruction.
 39. A computer-readable medium containing executableinstructions that, when executed by a computing apparatus, cause thecomputing apparatus to: (a) receive an image of human-comprehensiblecontent in a paper version of a document, wherein thehuman-comprehensible content is sufficient to locate a digital versionof the document; (b) locate a digital version of the document using theimage of human-comprehensible content; (c) receive an annotation of thepaper version of the document; (d) apply a recognition routine to theannotation to determine if the annotation includes anelectronically-executable instruction; and (e) store the annotation inassociation with the digital version of the document.
 40. Thecomputer-readable medium of claim 39, further comprisingcomputer-executable instructions that, when executed by the computingapparatus, cause the computing apparatus to use a spatial relationshipamong the human-comprehensible content to identify the location of theannotation in the document.
 41. The computer-readable medium of claim39, wherein the computer-executable instructions further cause thecomputing apparatus to execute the electronically-executable instructiondetermined to be in the annotation.
 42. The computer-readable medium ofclaim 39, wherein the computer-executable instructions cause thecomputing apparatus to display the annotation in the digital version ofthe document at approximately the same location as the annotationappears on the paper version of the document.
 43. The computer-readablemedium of claim 39, wherein the computer-executable instructions causethe computing apparatus to display the annotation as a footnote in thedigital version of the document with a footnote indicator appearing atapproximately the same location as the annotation appears on the paperversion of the document.
 44. The computer-readable medium of claim 39,wherein the computer-executable instructions further cause the computingapparatus to print the digital version of the document with theannotation.
 45. The computer-readable medium of claim 39, wherein theelectronically-executable instruction within the annotation starts withone or more predefined characters or words that are recognized asindicating the executable instruction.
 46. A computer-implemented methodfor processing an annotation of a document, comprising: (a) recording anannotation of a paper version of a document; (b) obtaining an image ofhuman-comprehensible content in the document, wherein thehuman-comprehensible content is sufficient to identify a digital versionof the document and further identify a location of the annotation in thedocument; (c) locating a digital version of the document using the imageof human-comprehensible content; (d) identifying the location of theannotation in the document using the image of human-comprehensiblecontent; (e) determining if the annotation includes anelectronically-executable instruction; and (f) storing the annotationand location in association with the digital version of the document.47. The computer-implemented method of claim 46, wherein the image ofhuman-comprehensible content is obtained at approximately the same timethe annotation is recorded.
 48. The computer-implemented method of claim46, further comprising, at the location of the annotation in thedocument, augmenting the digital version of the document with the storedannotation.
 49. The computer-implemented method of claim 48, whereinaugmenting the digital version of the document with the annotationincludes causing the annotation to appear in the digital version of thedocument at approximately the same location as the annotation appears onthe paper version of the document.
 50. The computer-implemented methodof claim 48, wherein augmenting the digital version of the document withthe annotation includes causing the annotation to appear as a footnotein the digital version of the document with a footnote indicatorappearing at approximately the same location as the annotation appearson the paper version of the document.
 51. The computer-implementedmethod of claim 48, further comprising printing the augmented digitalversion of the document.
 52. The computer-implemented method of claim46, wherein the electronically-executable instruction starts with one ormore predefined characters or words that are recognized as indicatingthe executable instruction.